Apparatus and method for client-side compositing of video streams

ABSTRACT

The present invention relates to an apparatus and method for client-side compositing of video streams. The method includes receiving, by the video display device, a multiplexed data stream from a remote video server. The multiplexed data stream includes substreams, where the substreams includes a descriptor substream, at least one media substream, and a compositing-instruction substream. The method further includes demultiplexing, by the video display device, the multiplexed data stream into the substreams according to the descriptor substream and displaying, by the video display device, the at least one media substream on a display screen according to the compositing-instruction substream, where the compositing-instruction substream includes instructions on a composition of the at least one media substream.

BACKGROUND

Many video distribution systems perform all video compositing at theserver. However, such server-side compositing often results in a finalcomposited video that is not well tailored to the client's device or theclient's personal preferences.

For example, when a broadcast network broadcasts a baseball game, theclient device often displays one or two information boxes containing thecurrent status of the game (current score, current ball and strikecount, etc.). These on-screen displays (OSDs) are intended to bedisplayed in the far upper-left and far upper-right corners of thedisplay screen. When watching the baseball game on a traditional squaretelevision set, the OSDs are displayed in this location. However, whenwatching the baseball game on a newer wide-screen television set, theOSDs may not be located in the far upper-left and far upper-rightcorners of the display screen. Rather, the OSDs may be displayed closertowards the center of the display screen. This behavior occurs becausethe video image is intentionally composited at the video server suchthat the OSDs are visible on all types of display screens. Becausewide-screen television sets have more horizontal display area thantraditional square television sets, the OSDs appear to be closer to thecenter of the wide-screen display. Hence, it can be appreciated thatoften the best way to display a video stream is a function of the typeof display screen of the client device. In another example, a user maywish to not view the OSDs sent by the broadcaster. Unfortunately,conventional composite-at-the-server methods do not allow a user tocontrol whether an OSD is displayed. In yet another example, a user maywant to switch formats of a movie depending on what type of clientdevice he/she is using. For example, if a movie is filmed in wide-screenformat and the user has a traditional square television set, the usermay want the ability to choose whether to display the movie inletter-box format or square format. Unfortunately, changing thecomposition of the display images according to the preference of a useris limited in conventional systems.

SUMMARY

The present invention relates to an apparatus and method for client-sidecompositing of video streams.

The method includes receiving, by the video display device, amultiplexed data stream from a remote video server. The multiplexed datastream includes substreams, where the substreams includes a descriptorsubstream, at least one media substream, and a compositing-instructionsubstream. The method further includes demultiplexing, by the videodisplay device, the multiplexed data stream into the substreamsaccording to the descriptor substream and displaying, by the videodisplay device, the at least one media substream on a display screenaccording to the compositing-instruction substream, where thecompositing-instruction substream includes instructions on a compositionof the at least one media substream.

The displaying step may display the at least one media substream as anon-screen display or a picture-in-picture in an area of the displayscreen indicated by the compositing-instruction substream.

The method may further include receiving, by the video display device, afirst instruction command from a user. The first instruction commandchanges a composition of the at least one media substream to acomposition different from the composition indicated by thecompositing-instruction substream, where the displaying step displaysthe at least one media substream according to the changed composition.Also, the first instruction command may indicate a removal of a mediasubstream included in the multiplexed data stream. Also, the firstinstruction command may indicate a change of at least one of a displaysize, display color, display shape, or display location of the at leastone media substream. Furthermore, the first instruction command mayindicate an addition of a media substream to the multiplexed datastream.

In another embodiment, the multiplexed data stream may include aplurality of media substreams. Each of the plurality of media substreamsincludes the same video data according to different display aspectratios. The display step displays at least one of the plurality of mediasubstreams that has a display aspect ratio that is closest to the videodisplay device based on the compositing-instruction substream.

Embodiments of the present invention also include a method forclient-side compositing of media streams by a video server. The methodincludes generating, by the video server, a compositing-instructionsubstream. The compositing-instruction substream includes instructionson a composition of at least one media substream. The method furtherincludes multiplexing, by the video server, a descriptor substream, theat least one media substream, and the compositing-instruction substreamand transmitting, by the video server, the multiplexed data stream to bedisplayed on a display screen according to the compositing-instructionsubstream.

The compositing-instruction substream may indicate an area of thedisplay screen to display the at least one media substream as anon-screen display or picture-in-picture.

The method may further include receiving, by the video server, a firstinstruction command from the video display device. The first instructioncommand changes a composition of the at least one media substream to acomposition different from the composition indicated by thecompositing-instruction substream, where the generating step generatesan updated compositing-instruction substream indicating the changedcomposition. The first instruction command may indicate a removal of amedia substream included in the transmitted multiplexed data stream.Also, the first instruction command may indicate a change of at leastone of a display size, display color, display shape, or display locationof the at least one media substream included in the multiplexed datastream. Also, the first instruction command may indicate an addition ofa media substream to the multiplexed data stream.

In another embodiment, the multiplexed data stream includes a pluralityof media substreams. Each of the plurality of media substreams includesthe same video data according to different display aspect ratios. Thecompositing-instruction substream may include instructions for a videodisplay device to select one of the plurality of media substreams thathas a display aspect ratio that is closest to the display device.

Embodiments of the present invention also include a video display devicefor client-side compositing of media streams. The video display deviceincludes a demultiplexer configured to demultiplex a multiplexed datastream into substreams. The substreams includes a descriptor sub-stream,at least one media substream, and a compositing-instruction substream.The compositing-instruction substream includes instructions on acomposition of the at least one media substream. The video displaydevice also includes a controller configured to display the at least onemedia substream on a display screen according to thecompositing-instruction substream. The controller may display the atleast one media substream as an on-screen display or apicture-in-picture in an area of the display screen indicated by thecompositing-instruction substream.

Also, the video display device may be configured to receive a firstinstruction command from a user. The first instruction command changes acomposition of the at least one media substream to a compositiondifferent from the composition indicated by the compositing-instructionsubstream, where the controller displays the at least one mediasubstream according to the changed composition. The first instructioncommand may indicate a removal of a media substream included in themultiplexed data stream. The first instruction command may indicate achange of at least one of a display size, display color, display shape,or display location of the at least one media substream. Also, the firstinstruction command may indicate an addition of a media substream to themultiplexed data stream.

According to another embodiment, the multiplexed data stream includes aplurality of media substreams. Each of the plurality of media substreamsincludes the same video data according to different display aspectratios, and the controller displays at least one of the plurality ofmedia substreams that has a display aspect ratio that is closest to thevideo display device based on the compositing-instruction substream.

Embodiments of the present invention also include a video server forclient-side compositing of media streams. The video server includes acontroller configured to generate a compositing-instruction substream.The compositing-instruction substream includes instructions on acomposition of at least one media substream. The video server alsoincludes a multiplexer configured to multiplex a descriptor substream,the at least one media substream, and the compositing-instructionsubstream, and configured to transmit the multiplexed data stream to bedisplayed on a display screen according to the compositing-instructionsubstream.

The compositing-instruction substream may indicate an area of thedisplay screen to display the at least one media substream as anon-screen object or picture-in-picture.

Also, the video server is configured to receive a first instructioncommand from the video display device. The first instruction commandchanges a composition of the at least one media substream to acomposition different from the composition indicated by thecompositing-instruction substream, and the controller generates anupdated compositing-instruction substream indicating the changedcomposition. The first instruction command may indicate a removal of amedia substream included in the transmitted multiplexed data stream.Also, the first instruction command may indicate a change of at leastone of a display size, display color, display shape, and displaylocation of the at least one media substream included in the multiplexeddata stream. Furthermore, the first instruction command may indicate anaddition of a media substream to the multiplexed data stream.

In another embodiment, the multiplexed data stream includes a pluralityof media substreams. Each of the plurality of media substreams includesthe same video data according to different display aspect ratios. Thecompositing-instruction substream includes instructions for a videodisplay device to select one of the plurality of media substreams thathas a display aspect ratio that is closest to the display device.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will become more fully understood from the detaileddescription given herein below and the accompanying drawings, whereinlike elements are represented by like reference numerals, which aregiven by way of illustration only and thus are not limiting of thepresent invention, and wherein:

FIG. 1 illustrates various components of a system for transmitting datastreams to video display devices according to an embodiment of thepresent invention;

FIG. 2 illustrates a video display device according to embodiments ofthe present invention;

FIG. 3 illustrates a video server according to embodiments of thepresent invention;

FIG. 4 illustrates a multiplexed data stream according to embodiments ofthe present invention;

FIG. 5 illustrates a multiplexed data stream having more than one mediasubstream according to embodiments of the present invention;

FIG. 6 illustrates an example of a client-mediated alteration of thedata stream according to embodiments of the present invention; and

FIG. 7 illustrates an example of a server-mediated alteration of thedata stream according to embodiments of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Various embodiments of the present invention will now be described morefully with reference to the accompanying drawings. Like elements on thedrawings are labeled by like reference numerals.

As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising,”, “includes” and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The present invention will now be described with reference to theattached figures. Various structures, systems and devices areschematically depicted in the drawings for purposes of explanation onlyand so as not to obscure the present invention with details that arewell known to those skilled in the art. Nevertheless, the attacheddrawings are included to describe and explain illustrative examples ofthe present invention. The words and phrases used herein should beunderstood and interpreted to have a meaning consistent with theunderstanding of those words and phrases by those skilled in therelevant art. To the extent that a term or phrase is intended to have aspecial meaning, i.e., a meaning other than that understood by skilledartisans, such a special definition will be expressly set forth in thespecification that directly and unequivocally provides the specialdefinition for the term or phrase.

Example embodiments provide a method and apparatus that allows the userto have more control over the displaying of video images on a displayscreen. For example, embodiments of the present invention composite atleast one of the media data streams on a video display device as opposedto the video server. For instance, as explained below, embodiments ofthe present invention make it possible to perform some or all thecompositing operations at the video display device (client-side),thereby enabling the client's device and preferences to be better takeninto account during the compositing operation, and thus resulting in afinal composited video that is tailored to the client's device andpreferences.

FIG. 1 illustrates various components of a system for transmitting datastreams to video display devices according to an embodiment of thepresent invention. For instance, FIG. 1 illustrates at least one videoserver 102 that transmits multiplexed data streams via a communicationinterface to at least one video display device 103 to be displayed on adisplay screen 104 of the video display device 103. The display screen104 may be included in the video display device 103 or a separate devicethat is connected to the video display device 103 via a wired and/orwireless connection. The multiplexed data streams may represent at leastone media stream. A media stream may be any type of media streamincluding but not limited to audio/visual (A/V) streams, andtext/graphic streams for On-Screen Displays (OSD). The media streamscover any number and type of codecs such as MPEG or HTML (e.g., theinvention is codec-neutral). The details of the multiplexed data streamsare further explained below.

The video display device 103 may include any type of system capable ofprocessing and displaying the transmitted data streams. For example, thevideo display device 103 may include a television 103-1 or a personalcomputer 103-3, or any combination thereof, for example. The television103-1 may be any type of conventional television set. Also, thetelevision 103-1 may be used in conjunction with any type of set-top box103-2 such as a cable box, a satellite receiver, or game console, forexample. The personal computer 103-3 may be any type of personalcomputer that has a processor, memory, and a display screen fordisplaying the transmitted multiplexed data streams. The video server102 is a computer based device (also called a ‘host’) dedicated todelivering video data.

The video server 102 transmits the multiplexed data streams to the atleast one video display device 103 through any type of communicationinterface such as wireless/wired internet connections, radiotransmissions (e.g., broadcast or satellite), or cable connections, forexample.

According to embodiments of the present invention, instead ofcompositing all the data streams at the video server 102, the videodisplay device 103 composites and displays at least one of the mediastreams according to a compositing-instruction substream, which isgenerated at the video server 102. For example, at any given moment, themultiplexed data stream that is transmitted from the video server 102 tothe video display device 103 includes one multiplex-descriptor substream(descriptor substream), zero or more media substreams, and zero or onecompositing-instruction substream. The descriptor substream is alwayspresent in the multiplexed data stream. However, at any given moment,the data stream may additionally include one or more media substreamsand/or the compositing-instruction substream. Thecompositing-instruction substream includes instructions on thecomposition of the one or more media substreams. In other words, thecompositing-instruction substream includes a set of instructions on howto composite one or more of the media substreams. Generally, themultiplexed data stream includes the compositing-instruction substreamwhen the video server 102 needs to change the compositing instructions,as further described below.

The descriptor substream describes the current contents of themultiplexed data stream, such as the number of each type of substreamcurrently present in the multiplexed data stream, and in the case ofmedia streams, the encoding used for each such substream. The supportedmedia encodings are, at a minimum, MPEG and HTML. However, embodimentsof the present invention emcompass any type of encoding/decoding.

FIG. 2 illustrates a video display device 103 according to embodimentsof the present invention. For example, the video display device 103includes a demultiplexer 201, a compositor 203, a controller 202, a userinterface 204, and the display screen 104, which is not shown in FIG. 2.The video display device 103 also includes other components that arewell known to one of ordinary skill in the art such as a tuner and/orantenna for receiving the transmitted multiplexed data streams, adecoder for decoding video data, and a memory/buffer for storing framesof the decoded video data, for example. The components of the videodisplay device 103 may be implemented in the television 103-1, theset-top box 103-2, and/or the personal computer 103-3 in a manner withinthe skill of someone with ordinary skill in the art.

Referring to FIG. 2, the video display device 103 receives themultiplexed data stream via a tuner or receiver antenna (not shown). Asindicated above, at any given point, the multiplexed data streamincludes one descriptor substream, zero or more media streams, and zeroor one compositing-instruction substream. The demultiplexer 201demultiplexes the multiplexed data stream into the substreams accordingto the descriptor substream. The descriptor substream includes, forexample, information on the number and type of substream that areincluded in the multiplexed data stream. The descriptor substream alsoincludes the necessary codec information to enable the receiver toproperly decode the media streams included in the multiplexed datastream. First, the demultiplexer 201 pulls out the multiplex descriptor,which the demultiplexer 201 knows is present in the multiplexed datastream. The demultiplexer 201 then reads the descriptor to find out theother contents contained in the multiplexed data stream. Thedemultiplexer 201 uses this information to demultiplex the rest of themultiplexed data stream. For example, the demultiplexer 201demultiplexes media substreams (AV₁ to AV_(N), ODS₁ to OSD_(N)) based onthe descriptor substream. Although the figure illustrates the mediastreams as A/V streams and text/graphic streams for OSDs, embodiments ofthe present invention cover any type of media stream. For instance, eachmedia stream has an encoding, and each is composited according to thecompositing instructions. The A/V and OSD categories are for explanatorypurposes only. Also, the demultiplexer 201 demultiplexes thecompositing-instruction substream INSTR from the multiplexed datastream, if this substream is present in the multiplexed data stream.Referring to FIG. 2, the substreams include any number media substreams(including zero). Although not shown, the video display device 103decodes the selected media substreams to generate frames that may bestored in a buffer memory in the video display device 103.

Next, the controller 203 displays the demultiplexed media substreams onthe display screen 104 of the video display device 103 according to thecompositing-instruction substream, which may be modified as explainedlater in the specification. For instance, the media substreams presentin the multiplexed data signal are received at the compositor 203 afterthey are demultiplexed. The controller 202 controls the compositor 203to composite the media substreams according to thecompositing-instruction substream. The compositing-instruction includesa set of composite instructions for the composition of the mediasubstreams. The composition may relate to the display size, displaycolor, display shape, and display location of the media substreams, forexample. However, the composite instructions may include any type ofcomposition instructions relating to the display of the mediasubstreams. For example, composition instruction may be predefined codesor syntaxes that include at least one byte in the multiplexed datastream. For example, the defined code (e.g., 1, 01, 101, etc.) may berecognized by the video display device 103 or the video server 102 aspertaining to a specific kind of instruction. Examples of specific kindsof instructions are discussed later in the specification.

After the media substreams are composited in the compositor 203according to the composition instructions, a graphics-display subsystem(not-shown) displays the composited display signal on the display screen104 of the video display device 103.

In addition, the video display device 103 includes a user interface 204configured to receive user input from an input device such as a computermouse or a remote controller, for example. In addition, a user mayprovide input directly to the video display device 103 by pressingcontrol buttons on the video display device 103 itself. The videodisplay device 103 may display a menu that allows a user to change thecomposition of the media substreams to a composition different from thecomposition indicated by the compositing-instruction substream.

Based on the user signal from the user, the video server 102 or thevideo display device 103 may perform the composite alteration. Thisfeature is explained later in the specification.

FIG. 3 illustrates a video server 102 according to embodiments of thepresent invention. The video server 102 includes a controller 302 and amultiplexer 301. The video server 102 also includes other componentsthat are well known in video servers. The controller 302 controls whichsubstreams are selected by the multiplexer 301 to be combined into themultiplexed data signal. For example, the controller 302 instructs themultiplexer 301 to multiplex any number of media substreams into themultiplexed data stream. Further, the video server 102 may include anencoder (not shown) to encode the media substreams using any knownencoding techniques.

The controller 302 generates the compositing-instruction substream INSTRand the descriptor substream to be multiplexed with the mediasubstreams. For instance, the controller 302 generates thecompositing-instruction substream INSTR based on the input of thedesigner of the data stream. As indicated earlier, thecompositing-instruction substream includes instructions on how the mediasubstreams are to be composited by the video display device 103. Thecontroller 302 generates the descriptor substream based on the number ofeach type of media substream to be included in the multiplexed datastream and the type of encoding to be applied. The video server 102transmits the multiplexed data stream to the video display device 103via the communication interface.

In addition, the video server 102 may include multiple media streams inthe multiplexed data stream, where each substream may or may notcorrespond to the same video content. In one embodiment of the presentinvention, each media substream may correspond to the same primary viewof a television program, but relate to different display aspect ratiosfor different sizes of display screens. Because different displayscreens have different aspect ratios, each video display device 103 mustcrop or stretch the media substream to the size of the display screen.However, to prevent undesirable cropping and stretching, the videoserver 102 may include a plurality of media substreams, where theplurality of media substreams include the same video data according todifferent display aspect ratios, and the video display device 103displays the media substream that has a display aspect ratio that isclosest to the display screen 104 of the video display device 103 basedon the composition-instruction substream. For example, the controller302 generates the compositing-instruction substream to include a set ofinstructions instructing the video display device 103 to select themedia substream that has the closest display aspect ratio of the displayscreen 104 of the video display device 103.

FIG. 4 illustrates a multiplexed data stream according to embodiments ofthe present invention. At time A, the multiplexed data stream includesan MPEG media substream 2 and the descriptor substream 1, which isalways present, per requirement. Suppose the MPEG media substream 2corresponds to the audio and video content of a baseball game. At timeB, the on-screen displays containing the current status of the game(current score, current ball and strike count, etc.) are first displayed(e.g., text/graphics substream 3 and text/graphics substream 4). At timeB, the multiplex data stream includes the composition-instructingsubstream 0, the multiplex descriptor 1, the MEPG media substream 2, andHTML substream 3 and HTML substream 4.

The compositing-instruction substream 0 may include instructions on howto display the HTML substream 3 and the HTML substream 4 as on-screendisplays in a display area of the display screen 104. For example, thecompositing-instruction substream may include instructions specifyingthe following: “composite HTML substream 3 in the upper-left corner,composite HTML substream 4 in the upper-right corner, and keepcompositing it there until further notice.” The above-identifiedcomposition instruction and any other composition instructionencompassed by the described embodiments may be represented by at leastone predefined code that includes at least one byte.

At time C, suppose the on-screen displays corresponding to the HTMLtext/graphic substream 3 and second HTML text/graphic substream 4 needto be updated due to a change in the score of the game. At this time,the multiplexed data stream includes the composite-instructing substream0, the multiplex descriptor 1, the MPEG media substream 2, and an HTMLmedia substream 3 including the updated information. Thecompositing-instruction substream at time C may include the followingcompositing instructions: “remove the previous HTML media substream 4from the composite, composite the new substream 3 in the upper-rightcorner, and keep compositing it there until further notice.”

FIG. 5 illustrates a multiplexed data stream having more than one mediasubstream according to embodiments of the present invention. Forinstance, image data associated with one MPEG substream may want to bedisplayed as a picture-in-picture of another MPEG substream. In order toaccomplish this, at time B, the video server 102 transmits the followingthe multiplexed data stream: the compositing-instruction substream 0,the descriptor substream 1, the MPEG media substream 2, and another MPEGmedia substream 3. The compositing-instruction substream includesinstructions on how to composite the MPEG media substream 3 with theMPEG media substream 2. For example, the compositing-instructionsubstream may include the following compositing instructions: “compositestream 3 in the lower-right corner.” The compositing-instructionsubstream at time C may be to stop compositing MPEG substream 3.

As indicated earlier, the video display device 103 may display a menuthat allows a user to change the composition of the media substreams toa composition different from the composition indicated by thecompositing-instruction substream. For example, the user provides hisselection via the input device or on the video display device 103itself, which is received by the user interface 204. This user commandis forward to the controller 202.

Based on the user command from the user, the video server 102 or thevideo display device 103 may perform the composite alteration. If thevideo server 102 performs the alteration, it may be referred to asserver-mediated viewer alteration. If the video display device 103performs the alteration, it may be referred to as client-mediated vieweralteration.

Referring to FIG. 2, in client-mediated viewer alteration, thecontroller 202 controls the compositor 203 to composite the mediasubstreams according to the user's preference. For example, if the userhas indicated not to display a particular media substream, thecompositor 203 does not composite that media substream into the finaldisplay signal. In addition, the controller 202 may control thecompositor 203 to composite the media streams in different displaylocations, display shapes, display sizes, display colors, etc. based onthe user's instruction command.

FIG. 6 illustrates an example of the client-mediated alteration of thedata stream according to embodiments of the present invention. Referringto FIG. 6, at time A, the multiplexed data stream includes thedescriptor substream 1 and one MPEG media substream 2. At time B, themultiplexed data stream includes the descriptor substream 1, the MPEGmedia substream 2, a compositing-instruction substream 0 and two HTMLsubstreams 3 and 4. The compositing-instruction substream 0 may includeinstructions on a display location for the two HTML substreams 3 and 4.At time C, the user uses the input device to pull down a display menuand selects “display only the MPEG video stream 2” or “turn off OSD.”Referring back to FIG. 2, the controller 202 then instructs thecompositor 203 to stop compositing all other media streams into thefinal display signal. However, the video server 102 continues totransmit these data streams in the multiplexed data stream.

In order to save network bandwidth, the controller 202 may transmit, ona user control channel, an instruction signal to the video server 102indicating which substreams the video server 102 should or should nottransmit. The user control channel may be the same upstream controlchannels that are used to carry the viewer inputs to the video server102.

In server-mediated alteration, the video server 102 performs the userinitiated alteration. For example, instead of the controller 202directing the compositor 203 to change the composition of the mediasubstreams, the controller 202 transmits on the user control channel aninstruction signal that includes the user's alterations to the videoserver 102. Then, the video server 102 makes the appropriate change tothe composition of the media substreams. For example, the video server102 may add additional media substreams indicated by the user. Also, thevideo server 102 may remove any type of media streams indicated by theuser. Also, the video server 102 may change the display location,display shape, display size, display color, etc. of any of the mediasubstreams.

FIG. 7 illustrates an example of a server-mediated alteration of thedata stream according to embodiments of the present invention. In FIG.7, at time C, the user uses the input device to pull down aserver-provided menu and selects “do not display HTML substreams 3 and4.” In response, the video display device 103 transmits an instructionsignal to the video server 102 via the user control channel. Referringto FIG. 3, the controller 301 generates a composite-instructionsubstream that includes a set of instructions not to display the textsubstreams 3 and 4 such as “stop compositing media substreams 3 and 4.”The composite instructions arrive at the client at time D in FIG. 7.Additionally, the video server 102 may stop transmitting the mediasubstreams 3 and 4. As a result, the video server 102 does not transmitthis media substream in the multiplexed data stream.

Variations of the example embodiments of the present invention are notto be regarded as a departure from the spirit and scope of the exampleembodiments of the invention, and all such variations as would beapparent to one skilled in the art are intended to be included withinthe scope of this invention.

1. A method for client-side compositing of media streams by a videodisplay device, the method comprising: receiving, by the video displaydevice, a multiplexed data stream from a remote video server, themultiplexed data stream including substreams, the substreams including adescriptor substream, at least one media substream, and acompositing-instruction substream; demultiplexing, by the video displaydevice, the multiplexed data stream into the substreams according to thedescriptor substream; displaying, by the video display device, the atleast one media substream on a display screen according to thecompositing-instruction substrearn, the compositing-instructionsubstream including instructions on a composition of the at least onemedia substream.
 2. The method of claim 1, wherein the displaying stepdisplays the at least one media substream as an on-screen display or apicture-in-picture in an area of the display screen indicated by thecompositing-instruction substream.
 3. The method of claim 1, furthercomprising: receiving, by the video display device, a first instructioncommand from a user, the first instruction command changing acomposition of the at least one media substream to a compositiondifferent from the composition indicated by the compositing-instructionsubstream, wherein the displaying step displays the at least one mediasubstream according to the changed composition.
 4. The method of claim3, wherein the first instruction command indicates a removal of a mediasubstream included in the multiplexed data stream.
 5. The method ofclaim 3, wherein the first instruction command indicates a change of atleast one of a display size, display color, display shape, or displaylocation of the at least one media substream.
 6. The method of claim 3,wherein the first instruction command indicates an addition of a mediasubstream to the multiplexed data stream.
 7. The method of claim 1,wherein the multiplexed data stream includes a plurality of mediasubstreams, each of the plurality of media substreams includes samevideo data according to different display aspect ratios, the displaystep displays at least one of the plurality of media substreams that hasa display aspect ratio that is closest to the video display device basedon the compositing-instruction substream.
 8. A method for client-sidecompositing of media streams by a video server, the method comprising:generating, by the video server, a compositing-instruction substream,the compositing-instruction substream including instructions on acomposition of at least one media substream; multiplexing, by the videoserver, a descriptor substream, the at least one media substream, andthe compositing-instruction substream; transmitting, by the videoserver, the multiplexed data stream to be displayed on a display screenaccording to the compositing-instruction substream.
 9. The method ofclaim 8, wherein the compositing-instruction substream indicates an areaof the display screen to display the at least one media substream as anon-screen display or picture-in-picture.
 10. The method of claim 8,further comprising: receiving, by the video server, a first instructioncommand from the video display device, the first instruction commandchanging a composition of the at least one media substream to acomposition different from the composition indicated by thecompositing-instruction substream, wherein the generating step generatesan updated compositing-instruction substream indicating the changedcomposition.
 11. The method of claim 10, wherein the first instructioncommand indicates a removal of a media substream included in thetransmitted multiplexed data stream.
 12. The method of claim 10, whereinthe first instruction command indicates a change of at least one of adisplay size, display color, display shape, or display location of theat least one media substream included in the multiplexed data stream.13. The method of claim 10, wherein the first instruction commandindicates an addition of a media substream to the multiplexed datastream.
 14. The method of claim 8, wherein the multiplexed data streamincludes a plurality of media substreams, each of the plurality of mediasubstreams includes same video data according to different displayaspect ratios, the compositing-instruction substream includinginstructions for a video display device to select one of the pluralityof media substreams that has a display aspect ratio that is closest tothe display device.
 15. A video display device for client-sidecompositing of media streams, the video display device comprising: ademultiplexer configured to demultiplex a multiplexed data stream intosubstreams, the substreams including a descriptor sub-stream, at leastone media substream, and a compositing-instruction substream, thecompositing-instruction substream including instructions on acomposition of the at least one media substream; a controller configuredto display the at least one media substream on a display screenaccording to the compositing-instruction substream.
 16. The videodisplay device of claim 15, wherein the controller displays the at leastone media substream as an on-screen display or a picture-in-picture inan area of the display screen indicated by the compositing-instructionsubstream.
 17. The video display device of claim 15, wherein the videodisplay device is configured to receive a first instruction command froma user, the first instruction command changing a composition of the atleast one media substream to a composition different froth thecomposition indicated by the compositing-instruction substream, whereinthe controller displays the at least one media substream according tothe changed composition.
 18. The video display device of claim 17,wherein the first instruction command indicates a removal of a mediasubstream included in the multiplexed data stream.
 19. The video displaydevice of claim 17, wherein the first instruction command indicates achange of at least one of a display size, display color, display shape,or display location of the at least one media substream.
 20. The videodisplay device of claim 17, wherein the first instruction commandindicates an addition of a media substream to the multiplexed datastream.
 21. The video display device of claim 15, wherein themultiplexed data stream includes a plurality of media substreams, eachof the plurality of media substreams includes same video data accordingto different display aspect ratios, and the controller displays at leastone of the plurality of media substreams that has a display aspect ratiothat is closest to the video display device based on thecompositing-instruction substream.
 22. A video server for client-sidecompositing of media streams, the video server comprising: a controllerconfigured to generate a compositing-instruction substream, thecompositing-instruction substream including instructions on acomposition of at least one media substream; a multiplexer configured tomultiplex a descriptor substream, the at least one media substream, andthe compositing-instruction substream; and the video server configuredto transmit the multiplexed data stream to be displayed on a displayscreen according to the compositing-instruction substream.
 23. The videoserver of claim 22, wherein the compositing-instruction substreamindicates an area of the display screen to display the at least onemedia substream as an on-screen object or picture-in-picture.
 24. Thevideo server of claim 22, wherein the video server is configured toreceive a first instruction command from the video display device, thefirst instruction command changing a composition of the at least onemedia substream to a composition different from the compositionindicated by the compositing-instruction substream, and the controllergenerates an updated compositing-instruction substream indicating thechanged composition.
 25. The video server of claim 24, wherein the firstinstruction command indicates a removal of a media substream included inthe transmitted multiplexed data stream.
 26. The video server of claim24, wherein the first instruction command indicates a change of at leastone of a display size, display color, display shape, and displaylocation of the at least one media substream included in the multiplexeddata stream.
 27. The video server of claim 24, wherein the firstinstruction command indicates an addition of a media substream to themultiplexed data stream.
 28. The video server of claim 22, wherein themultiplexed data stream includes a plurality of media substreams, eachof the plurality of media substreams includes the same video dataaccording to different display aspect ratios, thecompositing-instruction substream including instructions for a videodisplay device to select one of the plurality of media substreams thathas a display aspect ratio that is closest to the display device.